{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "862760b0",
   "metadata": {},
   "source": [
    "# Tool Selection Evaluator\n",
    "\n",
    "### Getting Started\n",
    "This sample demonstrates how to use tool selection evaluator on agent data. The supported input formats include:\n",
    "- simple data such as strings and `dict` describing tool calls;\n",
    "- user-agent conversations in the form of list of agent messages. \n",
    "\n",
    "Before you begin:\n",
    "```bash\n",
    "pip install azure-ai-evaluation\n",
    "```\n",
    "Set these environment variables with your own values:\n",
    "1) **MODEL_DEPLOYMENT_NAME** - The deployment name of the model for this AI-assisted evaluator, as found under the \"Name\" column in the \"Models + endpoints\" tab in your Azure AI Foundry project.\n",
    "2) **AZURE_OPENAI_ENDPOINT** - Azure Open AI Endpoint to be used for evaluation.\n",
    "3) **AZURE_OPENAI_API_KEY** - Azure Open AI Key to be used for evaluation.\n",
    "4) **AZURE_OPENAI_API_VERSION** - Azure Open AI Api version to be used for evaluation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31a75ee6",
   "metadata": {},
   "source": [
    "The Tool Selection evaluator assesses the appropriateness and efficiency of tool choices made by an AI agent by examining:\n",
    "- Relevance of selected tools to the conversation\n",
    "- Completeness of tool selection according to task requirements\n",
    "- Efficiency in avoiding unnecessary or redundant tools\n",
    "\n",
    "The evaluator uses a binary scoring system:\n",
    "\n",
    "    - Score 0 (Fail): Tools selected are irrelevant, incorrect, or missing essential tools\n",
    "    - Score 1 (Pass): All needed tools are selected, even if there are redundant tools\n",
    "\n",
    "This evaluation focuses on measuring whether the right tools were chosen for the task, regardless of how those tools were executed or their parameter correctness."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "353f2304",
   "metadata": {},
   "source": [
    "Tool Selection requires following input:\n",
    "- Query - This can be a single query or a list of messages(conversation history with agent). The original task request from the user.\n",
    "- Tool Calls - Tool Call(s) made by Agent to answer the query. Optional - if response has tool calls, if not provided evaluator will look for tool calls in response.\n",
    "- Response - (Optional) Response from Agent (or any GenAI App). This can be a single text response or a list or messages generated as part of Agent Response. If tool calls are not provided, Tool Selection Evaluator will look at response for tool calls.\n",
    "- Tool Definitions - Tool(s) definition used by Agent to answer the query. Required to understand available tools and their purposes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75125e94",
   "metadata": {},
   "source": [
    "### Initialize Tool Selection Evaluator\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5f31144",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from azure.ai.evaluation._evaluators._tool_selection import _ToolSelectionEvaluator\n",
    "from azure.ai.evaluation import AzureOpenAIModelConfiguration\n",
    "from pprint import pprint\n",
    "\n",
    "model_config = AzureOpenAIModelConfiguration(\n",
    "    azure_endpoint=os.environ[\"AZURE_OPENAI_ENDPOINT\"],\n",
    "    api_key=os.environ[\"AZURE_OPENAI_API_KEY\"],\n",
    "    api_version=os.environ[\"AZURE_OPENAI_API_VERSION\"],\n",
    "    azure_deployment=os.environ[\"MODEL_DEPLOYMENT_NAME\"],\n",
    ")\n",
    "\n",
    "\n",
    "tool_selection = _ToolSelectionEvaluator(model_config=model_config)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb435367",
   "metadata": {},
   "source": [
    "### Samples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "344fe68e",
   "metadata": {},
   "source": [
    "#### Evaluating Single Tool Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcb71257",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "tool_call = {\n",
    "    \"type\": \"tool_call\",\n",
    "    \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"arguments\": {\"location\": \"Seattle\"},\n",
    "}\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"id\": \"fetch_weather\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"send_email\",\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "response = tool_selection(query=query, tool_calls=tool_call, tool_definitions=tool_definitions)\n",
    "pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e078b1fa",
   "metadata": {},
   "source": [
    "#### Multiple Tool Selections for Complex Task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "322efe28",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"arguments\": {\"location\": \"Seattle\"},\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "        \"name\": \"send_email\",\n",
    "        \"arguments\": {\n",
    "            \"recipient\": \"user@example.com\",\n",
    "            \"subject\": \"Weather Information for Seattle\",\n",
    "            \"body\": \"Weather data will be included here.\",\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"id\": \"fetch_weather\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"send_email\",\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"get_calendar\",\n",
    "        \"name\": \"get_calendar\",\n",
    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "response = tool_selection(query=query, tool_calls=tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19b3c8de",
   "metadata": {},
   "source": [
    "#### Tool Calls passed as part of `Response` (common for agent case)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "629f3cc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Can you send me an email with weather information for Seattle?\"\n",
    "response = [\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:35Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "                \"name\": \"fetch_weather\",\n",
    "                \"arguments\": {\"location\": \"Seattle\"},\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:37Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_CUdbkBfvVBla2YP3p24uhElJ\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [{\"type\": \"tool_result\", \"tool_result\": {\"weather\": \"Rainy, 14°C\"}}],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:38Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"tool_call\",\n",
    "                \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "                \"name\": \"send_email\",\n",
    "                \"arguments\": {\n",
    "                    \"recipient\": \"your_email@example.com\",\n",
    "                    \"subject\": \"Weather Information for Seattle\",\n",
    "                    \"body\": \"The current weather in Seattle is rainy with a temperature of 14°C.\",\n",
    "                },\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:41Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"tool_call_id\": \"call_iq9RuPxqzykebvACgX8pqRW2\",\n",
    "        \"role\": \"tool\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"tool_result\", \"tool_result\": {\"message\": \"Email successfully sent to your_email@example.com.\"}}\n",
    "        ],\n",
    "    },\n",
    "    {\n",
    "        \"createdAt\": \"2025-03-26T17:27:42Z\",\n",
    "        \"run_id\": \"run_zblZyGCNyx6aOYTadmaqM4QN\",\n",
    "        \"role\": \"assistant\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"type\": \"text\",\n",
    "                \"text\": \"I have successfully sent you an email with the weather information for Seattle. The current weather is rainy with a temperature of 14°C.\",\n",
    "            }\n",
    "        ],\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"get_calendar\",\n",
    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_selection(query=query, response=response, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "682839b8",
   "metadata": {},
   "source": [
    "#### Query as Conversation History (List of Messages)\n",
    "The evaluator also supports query as a list of messages representing conversation history. This helps determine if the Agent selected appropriate tools based on the conversation context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "218ade7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Query as conversation history instead of a single string\n",
    "query_as_conversation = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that can fetch weather information and send emails.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\", \n",
    "        \"content\": \"Hi, can you check the weather in Seattle for me?\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"Actually, could you also send me an email with that weather information to john@example.com?\"\n",
    "    }\n",
    "]\n",
    "\n",
    "tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_weather_123\",\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"arguments\": {\"location\": \"Seattle\"},\n",
    "    },\n",
    "    {\n",
    "        \"type\": \"tool_call\", \n",
    "        \"tool_call_id\": \"call_email_456\",\n",
    "        \"name\": \"send_email\",\n",
    "        \"arguments\": {\n",
    "            \"recipient\": \"john@example.com\",\n",
    "            \"subject\": \"Weather Information for Seattle\",\n",
    "            \"body\": \"Here is the weather information you requested.\"\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"get_calendar\",\n",
    "        \"description\": \"Retrieves calendar events for a specified date range.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"start_date\": {\"type\": \"string\", \"description\": \"Start date for calendar events.\"},\n",
    "                \"end_date\": {\"type\": \"string\", \"description\": \"End date for calendar events.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_selection(query=query_as_conversation, tool_calls=tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "406bc90c",
   "metadata": {},
   "source": [
    "#### Example of Poor Tool Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2c451e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "# Using irrelevant tool for the task\n",
    "poor_tool_calls = [\n",
    "    {\n",
    "        \"type\": \"tool_call\",\n",
    "        \"tool_call_id\": \"call_calendar_123\",\n",
    "        \"name\": \"get_calendar\",\n",
    "        \"arguments\": {\"start_date\": \"2025-01-01\", \"end_date\": \"2025-01-31\"},\n",
    "    },\n",
    "]\n",
    "\n",
    "# This should score poorly as get_calendar is not relevant for weather queries\n",
    "result = tool_selection(query=query, tool_calls=poor_tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab4a46c7",
   "metadata": {},
   "source": [
    "#### Response as String (str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e34bec52",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What's the weather in New York?\"\n",
    "\n",
    "# Response as a simple string (not commonly used, but supported by the API)\n",
    "# When using string response, tool_calls should be provided separately\n",
    "response_str = \"The current weather in New York is sunny with a temperature of 22°C.\"\n",
    "\n",
    "tool_calls = {\n",
    "    \"type\": \"tool_call\",\n",
    "    \"tool_call_id\": \"call_ny_weather_789\",\n",
    "    \"name\": \"fetch_weather\",\n",
    "    \"arguments\": {\"location\": \"New York\"},\n",
    "}\n",
    "\n",
    "tool_definitions = [\n",
    "    {\n",
    "        \"name\": \"fetch_weather\",\n",
    "        \"description\": \"Fetches the weather information for the specified location.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The location to fetch weather for.\"}},\n",
    "        },\n",
    "    },\n",
    "    {\n",
    "        \"name\": \"send_email\",\n",
    "        \"description\": \"Sends an email with the specified subject and body to the recipient.\",\n",
    "        \"parameters\": {\n",
    "            \"type\": \"object\",\n",
    "            \"properties\": {\n",
    "                \"recipient\": {\"type\": \"string\", \"description\": \"Email address of the recipient.\"},\n",
    "                \"subject\": {\"type\": \"string\", \"description\": \"Subject of the email.\"},\n",
    "                \"body\": {\"type\": \"string\", \"description\": \"Body content of the email.\"},\n",
    "            },\n",
    "        },\n",
    "    },\n",
    "]\n",
    "\n",
    "result = tool_selection(query=query, response=response_str, tool_calls=tool_calls, tool_definitions=tool_definitions)\n",
    "pprint(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc857725",
   "metadata": {},
   "source": [
    "#### Tool Definition as Single Dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d94822e1",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"How is the weather in Seattle?\"\n",
    "\n",
    "tool_definition_dict = {\n",
    "    \"name\": \"get_weather\",\n",
    "    \"description\": \"Get the current weather for a location\",\n",
    "    \"parameters\": {\n",
    "        \"type\": \"object\",\n",
    "        \"properties\": {\n",
    "            \"location\": {\"type\": \"string\", \"description\": \"The city and state, e.g. San Francisco, CA\"},\n",
    "            \"unit\": {\"type\": \"string\", \"enum\": [\"celsius\", \"fahrenheit\"]}\n",
    "        },\n",
    "        \"required\": [\"location\"]\n",
    "    }\n",
    "}\n",
    "\n",
    "tool_calls_dict = {\n",
    "    \"type\": \"tool_call\",\n",
    "    \"tool_call_id\": \"call_abc123\",\n",
    "    \"name\": \"get_weather\",\n",
    "    \"arguments\": {\"location\": \"Seattle, WA\", \"unit\": \"celsius\"}\n",
    "}\n",
    "\n",
    "result = tool_selection(query=query, tool_definitions=tool_definition_dict, tool_calls=tool_calls_dict)\n",
    "\n",
    "pprint(result)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_agent_evaluator_prp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
